Image analysis apparatus, image analysis method, and non-transitory computer-readable storage medium storing image analysis program

ABSTRACT

An image analysis apparatus includes one or more processors configured to execute (a) acquiring, from a measurement device, spectral images for a plurality of wavelengths, obtained by imaging a measurement target, (b) acquiring a target range in each of the spectral images, (c) performing multivariate analysis of each pixel based on a gradation value of the pixel for each wavelength in the target range, (d) generating an analysis image including an analysis result of the multivariate analysis for each pixel in the target range, and (e) storing the generated analysis image into a memory.

The present application is based on, and claims priority from JPApplication Serial Number 2020-170963, filed Oct. 9, 2020, thedisclosure of which is hereby incorporated by reference herein in itsentirety.

BACKGROUND 1. Technical Field

The present disclosure relates to an image analysis apparatus, an imageanalysis method, and a non-transitory computer-readable storage mediumstoring an image analysis program.

2. Related Art

In the related art, there is an image analysis apparatus that images atarget object with a spectral camera and analyzes the spectral imageobtained through the imaging (refer to, for example, JP-A-2014-153066).

An image analysis apparatus (measurement device) disclosed inJP-A-2014-153066 includes an optical sensor section capable of capturing16-band spectral images. In this measurement device, three-band spectralimages corresponding to, for example, red (R), blue (B), and green (G)are captured by the optical sensor section, and these images arecombined to generate a real-time image that is displayed on a displaysection. When a part of the real-time image is designated by a user, forexample, a range of 10×10 including a designated position is set as acolorimetric region, and 16-band spectral images of only thecolorimetric region are captured. An average light amount value of allpixels of the spectral image in the colorimetric region obtained from acolorimetric result is calculated for each wavelength and output asspectrum data.

However, in JP-A-2014-153066, the spectrum data indicating a color ofthe colorimetric region is output as the measurement result, butintermediate data is not output. That is, a measured value of each pixelin the colorimetric region and spectrum data of each pixel are notoutput, and a user cannot determine whether or not the output spectrumdata is correct.

SUMMARY

An image analysis apparatus according to a first aspect of the presentdisclosure includes one or more processors configured to execute (a)acquiring, from a measurement device, spectral images for a plurality ofwavelengths, obtained by imaging a measurement target, (b) acquiring atarget range in each of the spectral images, (c) performing multivariateanalysis of each pixel based on a gradation value of the pixel for eachwavelength in the target range, (d) generating an analysis imageincluding an analysis result of the multivariate analysis for each pixelin the target range, and (e) storing the generated analysis image into amemory.

An image analysis method according to a second aspect of the presentdisclosure includes causing one or more processors to analyze spectralimages for a plurality of wavelengths, obtained by imaging a measurementtarget, in which the one or more processors function as a spectral imageacquisition section, a range acquisition section, an analysis section,and an analysis image generation section, the spectral image acquisitionsection executes a spectral image acquisition step of acquiring thespectral images for the plurality of wavelengths, obtained by imagingthe measurement target, the range acquisition section executes a rangeacquisition step of acquiring a target range in each of the spectralimages, an analysis section executes an analysis step of performingmultivariate analysis of each pixel based on a gradation value of thepixel for each wavelength in the target range, and an analysis imagegeneration section executes an analysis image generation step ofgenerating an analysis image including an analysis result of themultivariate analysis for each pixel in the target range.

A non-transitory computer-readable storage medium according to a thirdaspect of the present disclosure stores an image analysis program thatcan be read and executed by a computer and causes the computer tofunction as the above image analysis apparatus.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a schematic configuration of ananalysis system according to an embodiment of the present disclosure.

FIG. 2 is a diagram illustrating a functional configuration of aterminal control section of the present embodiment.

FIG. 3 is a flowchart illustrating an image analysis method in theanalysis system of the present embodiment.

FIG. 4 is a diagram illustrating an example of an analysis image of thepresent embodiment.

FIG. 5 is a diagram illustrating another example of the analysis imageof the present embodiment.

FIG. 6 is a diagram illustrating still another example of the analysisimage of the present embodiment.

FIG. 7 is a diagram illustrating an example of an analysis imagedisplayed on a display of the present embodiment.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, an embodiment according to the present disclosure will bedescribed.

FIG. 1 is a block diagram illustrating a schematic configuration of ananalysis system 1 of the present embodiment.

The analysis system 1 includes a measurement device 2 and a terminaldevice 3, which are configured to be able to communicate with eachother, and configure an image analysis apparatus according to thepresent disclosure.

In the analysis system 1, the measurement device 2 images measurementtarget light (incident light) from a measurement target X and outputs acaptured image to the terminal device 3. Consequently, a color image ofthe measurement target X is displayed on a display 31 (display section)coupled to the terminal device 3. The user operates the terminal device3 to designate a range in which the measurement is to be performed inthe color image displayed on the display 31, and the terminal device 3sets the designated range as the target range. The measurement device 2captures spectral images of the measurement target X for a plurality ofwavelengths. The terminal device 3 performs image analysis of a targetrange based on the captured spectral images for the plurality ofwavelengths, and outputs an analysis result and features in the targetrange obtained through the analysis result to the display 31.

Here, the image analysis in the present disclosure is a process ofanalyzing a component of each pixel based on a gradation value of eachpixel of a plurality of spectral images. The component to be analyzed isan index value when a feature of the measurement target X is determined,and differs depending on a feature to be determined. For example, when acolor feature in a target range of the measurement target X isdetermined, an index value of a color component of each pixel in thetarget range is analyzed. As index values of color components, colorsystem parameters such as RGB values, L*a*b* values, and xyz values maybe exemplified. When a material of the measurement target X isdetermined as a feature, a component included in the target range of themeasurement target X is analyzed as an index value. For example, when aningredient used for a dish or the like is determined as a feature, eachnutrient or a water content included in the target range is analyzed asan index value.

When a color is determined as in the former case, spectral images for aplurality of spectral wavelengths in a visible light region may beacquired as a plurality of spectral images, and, when a material or aphysical property of an object is determined, spectral images for aplurality of spectral wavelengths in an ultraviolet region or aninfrared region are acquired.

In the present embodiment, as an example, the analysis system 1 performsan analysis process for determining a color of a measurement target, andeach configuration will be described in detail below.

Structure of Measurement Device 2

As illustrated in FIG. 1, the measurement device 2 includes a spectralimaging section 21, a color imaging section 22, a light source 23, acommunication section 24, and a measurement control section 25.

The spectral imaging section 21 captures a spectral image of measurementtarget light from the measurement target X. The spectral imaging section21 includes an incident optical system 211, a spectral element 212, animage formation optical system 213, and an imaging element 214.

The incident optical system 211 guides measurement light from themeasurement target X to the spectral element 212. The incident opticalsystem 211 is configured with, for example, a plurality of lensesforming an image-side telecentric optical system, and converts main raysof the measurement light into parallel rays to be incident to thespectral element 212.

The spectral element 212 disperses light having a predeterminedwavelength from the measurement light and emits the light toward theimaging element 214. The spectral element 212 is not particularlylimited as long as the spectral element disperses light having apredetermined wavelength and can change a wavelength of light to bedispersed (spectral wavelength). For example, in the present embodiment,a Fabry-Perot etalon is used as the spectral element 212. TheFabry-Perot etalon is an element including a pair of reflective filmsfacing each other and a gap changing portion that changes a size of agap between the reflective films. In such a Fabry-Perot etalon, a sizeof the gap between the pair of reflective films is changed by the gapchanging portion, and thus it is possible to transmit light having awavelength corresponding to the gap from the measurement light.

As the spectral element 212, an acousto-optic tunable filter (AOTF), aliquid crystal tunable filter (LCTF), or the like may be used.

The image formation optical system 213 is configured with, for example,a plurality of lenses, and forms an image of light having a spectralwavelength transmitted through the spectral element 212 on the imagingelement 214.

The imaging element 214 is configured with, for example, a complementarymetal-oxide semiconductor (CMOS) sensor or a charge-coupled device (CCD)sensor, and receives light having a predetermined spectral wavelengththat has transmitted through the incident optical system 211, thespectral element 212, and the image formation optical system 213, andoutputs a spectral image.

The color imaging section 22 is a normal RGB camera that captures acolor image of the measurement target X.

That is, although not illustrated, the color imaging section 22 includesan incident optical system to which the measurement light is incident,an image formation optical system that forms an image of the measurementlight on an imaging element configured with a CCD, CMOS, or the like,and an imaging element, and, for example, RGB color filters arranged ina Bayer arrangement are arranged in the imaging element. Consequently,the color imaging section 22 captures a color image including agradation value of each color of red (R), green (G), and blue (B) as apixel value.

In the present embodiment, an example in which the color imaging section22 is provided is described, but the present disclosure is not limitedto this. For example, the spectral imaging section 21 captures aspectral image for one wavelength (for example, 680 nm) in the redwavelength region, a spectral image for one wavelength (for example, 550nm) in the green wavelength region, and a spectral image for onewavelength (for example, 400 nm) in the blue wavelength region. A colorimage may be synthesized by combining these three spectral images. Whensuch a synthesized image is used, the constituents of the color imagingsection 22 can be eliminated.

The light source 23 emits light toward the measurement target X, and themeasurement device 2 measures image light reflected by the measurementtarget X as the measurement light, that is, captures an image thereof.

The light source 23 emits uniform light with respect to the wavelengthregion of the spectral image captured by the spectral imaging section21. For example, the analysis system 1 of the present embodimentanalyzes a color of the measurement target X. In this case, the spectralimaging section 21 switches wavelengths of light dispersed by thespectral element 212 at predetermined wavelength intervals in thevisible light region of, for example, 400 nm to 700 nm, and captures aspectral image for each wavelength. Therefore, as the light source 23, awhite light source in which an amount of light at each wavelength in thevisible light region is substantially uniform may be used. When ananalysis process using the near-infrared region is performed in theanalysis system 1, it is necessary to capture a spectral image for eachwavelength in the near-infrared region, and thus a light source having asubstantially uniform amount of light for each wavelength in thenear-infrared region may be used as the light source 23.

The communication section 24 performs communication with the terminaldevice 3 or other external devices. A communication method of thecommunication section 24 is not particularly limited, and, for example,wired communication may be used, or wireless communication using awireless LAN or the like may be used.

The measurement control section 25 has various functions for controllingthe measurement device 2. Specifically, the measurement control section25 functions as a storage section 251, a spectral control section 252, acolor imaging control section 253, and a light source control section254. Each of these functional constituents is configured with acombination of an arithmetic circuit such as a CPU that functions as aprocessor provided on a circuit board, a storage circuit such as amemory, and various driver circuits. The processor in the presentdisclosure is a term that includes a CPU configured to execute asoftware program and a hardware circuit.

The storage section 251 stores various programs and various data forcontrolling the measurement device 2. For example, as the various data,for example, a drive table for controlling the spectral element 212 isrecorded. As described above, when the Fabry-Perot etalon is used as thespectral element 212 and the gap between the pair of reflective films iscontrolled by an electrostatic actuator, a relationship between aspectral wavelength and a value of a voltage applied to theelectrostatic actuator is recorded on the drive table. As the variousdata, information regarding a measurement wavelength when measuring themeasurement target X, for example, a measurement start wavelength, awavelength change interval, and a measurement end wavelength may berecorded. The storage section 251 records a spectral image or a colorimage captured by the spectral imaging section 21 or the color imagingsection 22.

The spectral control section 252 controls the spectral element 212 basedon the drive table stored in the storage section 251 to disperse lighthaving a desired spectral wavelength from the spectral element 212. Thespectral control section 252 controls the imaging element 214 of thespectral imaging section 21 to capture a spectral image.

The color imaging control section 253 controls the color imaging section22 to capture a color image.

The light source control section 254 controls turning on and off of thelight source 23, for example, based on a user's instruction.

Configuration of Terminal Device 3

As illustrated in FIG. 1, the terminal device 3 is a computer includinga display 31, an input section 32, a terminal communication section 33,a terminal storage section 34, and a terminal control section 35 as ahardware configuration. The terminal storage section 34 is a solid statedrive as a storage device. The terminal control section 35 has a CPU anda GPU as a processor, and a DRAM as a memory accessed from theprocessor. In the following disclosure, the storage device included inthe terminal storage section 34 and the memory included in the processormay be referred to as “memory” without particular distinction.

The display 31 is a display section according to the present disclosure,and various display devices such as a liquid crystal display and anorganic EL display may be used. In the present embodiment, an example inwhich the terminal device 3 includes the display 31 is described, butthe display 31 may be configured separately from the terminal device 3and coupled to the terminal device 3 in a communicable manner.

The input section 32 receives an operation from the user and outputs anoperation signal corresponding to the operation to the terminal controlsection 35. The input section 32 may be a touch panel integrally formedwith the display 31, or various input devices such as a mouse and akeyboard may be used.

The terminal communication section 33 is a terminal communicationinterface and communicates with the measurement device 2 or otherexternal devices. A communication method of the terminal communicationsection 33 is not particularly limited, and may be wired communicationor wireless communication.

The terminal storage section 34 records various data and variousprograms used in a calculation process executed by the terminal controlsection 35. Specifically, the terminal storage section 34 storesspectral images or color images for a plurality of spectral wavelengthsobtained from the measurement device 2, an analysis image generatedthrough the analysis process, an image analysis program for controllingthe analysis system 1, and the like.

The terminal control section 35 executes various functions by readingand executing various programs recorded in the terminal storage section34, and controls the terminal device 3 and the analysis system 1.

FIG. 2 is a diagram illustrating a functional configuration of theterminal control section 35.

The terminal control section 35 reads and executes the image analysisprogram recorded in the terminal storage section 34, and thus functionsas a display control section 351, a spectral image acquisition section352, a color image acquisition section 353, a range acquisition section354, an analysis section 355, a feature determination section 356, ananalysis image generation section 357, a selection acquisition section358, and the like as illustrated in FIG. 2.

The display control section 351 controls image display on the display31. Examples of the image to be displayed on the display 31 include acolor image captured by the color imaging section 22 of the measurementdevice 2, an analysis image generated by the analysis image generationsection 357, and an image of a feature of the measurement target Xdetermined by the feature determination section 356. In addition, anoperation screen for operating the analysis system 1, notificationinformation for notifying a user of an operating status of themeasurement device 2, and the like may be displayed.

The spectral image acquisition section 352 outputs a command for imaginga spectral image to the measurement device 2. Consequently, the spectralimaging section 21 of the measurement device 2 captures spectral imagesfor a plurality of spectral wavelengths and outputs the spectral imagesto the terminal device 3. The spectral image acquisition section 352acquires the spectral images output from the measurement device 2 andstores the spectral images into the terminal storage section 34.

The color image acquisition section 353 outputs a command for imaging acolor image to the measurement device 2. Consequently, the color imagingsection 22 of the measurement device 2 captures color images of themeasurement target X and outputs the color images to the terminal device3. The color image acquisition section 353 acquires the spectral imagesoutput from the measurement device 2. In the present embodiment, theacquired color images are displayed on the display 31 as real-timeimages by the display control section 351. That is, the color imagingsection 22 continuously captures images of the measurement target X at apredetermined sampling frequency until a stop command is input by theuser, and updates the real-time images displayed on the display 31.

The range acquisition section 354 acquires a range (target range) foranalyzing the measurement target X. Specifically, the user designates adesired target range for a color image of the measurement target Xdisplayed on the display 31, and thus the range acquisition section 354acquires a target range of a spectral image corresponding to the colorimage. The target range may be designated, for example, in apredetermined range centered on a point specified by the user, or may bea range freely set by the user, for example, by a drag operation.

The analysis section 355 specifies the target range acquired by therange acquisition section 354 in the spectral images for a plurality ofspectral wavelengths, and performs an analysis process based ongradation values of pixels in the target range of each spectral image.

Specifically, the analysis section 355 functions as a spectrum analysissection 355A, an index calculation section 355B, a classificationsection 355C, and a boundary setting section 355D.

The spectrum analysis section 355A calculates a spectral spectrum foreach pixel based on the gradation value of each pixel in the targetrange of a spectral image.

The index calculation section 355B analyzes an index value fordetermining a feature in the target range of the measurement target Xbased on the calculated spectral spectrum. Specifically, the indexcalculation section 355B calculates the index value for each pixel. Asdescribed above, since the spectral spectrum for each pixel, that is,the gradation value for each spectral wavelength is obtained, the indexcalculation section 355B performs a multivariate analysis process withthe gradation value for each of these spectral wavelengths as avariable. The multivariate analysis process is not particularly limited,and for example, various analysis methods such as principal componentanalysis and correspondence analysis may be used. An analysis method maybe designated by the user.

When the measurement device 2 measures spectral images for 16-bandspectral wavelengths, sixteen gradation values are acquired as variableparameters for each pixel. The spectrum analysis section 355A and theindex calculation section 355B perform multivariate analysis on thesixteen variable parameters to calculate the spectral spectrum and theindex value.

The classification section 355C associates pieces of pixel analysis datahaving similar index values in the target range and classifies the dataas the same group. As the group classification, for example, each pieceof pixel analysis data may be classified according to the number ofclassifications designated by the user, or analysis results from theindex calculation section 355B may be clustered such that the pixelanalysis data is automatically classified into a predetermined number ofgroups.

The boundary setting section 355D sets a boundary between the groupsclassified by the classification section 355C. As will be describedlater in detail, in the present embodiment, an analysis image in athree-axis coordinate system with each index value as an axis isgenerated based on three index values. The boundary setting section 355Dsets a boundary for separating each group when points corresponding tothe pixel analysis data are plotted on the three-axis coordinate systemof the analysis image. The boundary may be a plane or a curved surface.

The feature determination section 356 determines a feature of themeasurement target X in a pixel corresponding to the pixel analysis datain the target range based on the index value of each piece of pixelanalysis data. For example, in the present embodiment, a color of eachpixel in the target range is determined.

The feature determination section 356 may determine a feature of theentire target range. That is, the number of pixels included in the pixelgroup having the same feature is counted based on the feature of eachpixel included in the target range, and a feature of a pixel grouphaving the largest number of pixels is determined as a feature of thetarget range. The pixel group having the same feature may be a group ofpixels classified as the same group by the classification section.

The analysis image generation section 357 generates an analysis image inwhich each piece of pixel analysis data obtained through multivariateanalysis performed by the analysis section 355 is plotted on thethree-axis coordinate system. In this case, the analysis imagegeneration section 357 may represent the feature specified by thefeature determination section 356 in the analysis image. For example,when the feature determination section 356 determines “red”, each plotpoint is displayed as a red point, and when the feature determinationsection 356 determines “blue”, each plot point is displayed as a bluepoint.

When a predetermined plot point is selected by the user in the analysisimage displayed on the display 31, the selection acquisition section 358acquires the plot point.

Operation of Analysis System

FIG. 3 is a flowchart illustrating an image analysis method in theanalysis system 1 of the present embodiment.

In the analysis system 1 of the present embodiment, in image analysisfor the measurement target X, first, a user performs an input operationfor capturing an image of the measurement target X desired to bemeasured. Consequently, the terminal device 3 outputs a command signalfor giving a command for a color image capturing process to themeasurement device 2.

Consequently, the color imaging section 22 of the measurement device 2captures a color image of the measurement target X and outputs thecaptured color image to the terminal device 3. When the color image iscaptured by the measurement device 2, the terminal device 3 displays thecolor image on the display 31 (step S1). In the present embodiment, anexample in which a color image is captured by the color imaging section22 provided in the measurement device 2 is described, but a color imagemay be generated by combining spectral images captured by the spectralimaging section 21.

The terminal device 3 causes the display 31 to display guidanceinformation to the effect that the target range is designated. When theuser performs an input operation for designating a target range for thecolor image, the range acquisition section 354 acquires the designatedand input target range in the color image (step S2: range acquisitionstep). As described above, the target range may be a range within apredetermined distance to a point designated by the user, or may be arange designated by the user through a drag operation or the like. Aplurality of target ranges may be designated.

Thereafter, the terminal device 3 outputs a spectral imaging command forperforming a spectral imaging process to the measurement device 2.Consequently, the spectral imaging section 21 of the measurement device2 captures a spectral image for a preset spectral wavelength withrespect to a predetermined measurement wavelength region (step S3:spectral image acquisition step). For example, in the presentembodiment, in order to determine a color of the measurement target X,16-band spectral images at intervals of 20 nm are captured with thevisible light region of 700 nm to 400 nm as the measurement wavelengthregion.

Specifically, the spectral control section 252 of the measurementcontrol section 25 that has received the spectral imaging command readsthe drive table from the storage section 251 and sequentially applies adrive voltage corresponding to each wavelength at intervals of 20 nmfrom 700 nm to 400 nm to the gap changing portion of the spectralelement 212. Consequently, image light having each spectral wavelengthat intervals of 20 nm from 700 nm to 400 nm, transmitted through thespectral element 212, is sequentially imaged by the imaging element 214.

In step S3, a spectral image corresponding to the entire color imagecaptured in step S1 may be captured, and only a spectral image of thetarget range designated in step S2 in the color image captured in stepS1 may be captured. When a spectral image of the target range iscaptured, it is possible to suppress the pressure on the capacity of theterminal storage section 34 and to quickly capture the spectral image.

Thereafter, when the terminal device 3 receives the spectral image foreach spectral wavelength from the measurement device 2, the analysissection 355 performs an analysis step.

Specifically, the spectrum analysis section 355A calculates a spectralspectrum for each pixel, that is, the light intensity for each spectralwavelength, based on a gradation value of each pixel in the target rangeof each spectral image (step S4).

The index calculation section 355B performs a multivariate analysisprocess based on the calculated spectral spectrum, and thus calculatesan index value or a feature vector (step S5). Depending on processes, aparameter group required for the multivariate analysis process may bedetermined in advance based on a plurality of known spectral images.

As described above, the index value may be selected according to ananalysis target item of the measurement target X. For example, in thepresent embodiment, RGB values are calculated as index values in orderto determine a color of the target range. Here, RGB values areexemplified as index values used for color determination, but xyz valuesor L*a*b* values may be used. In the present embodiment, colordetermination is exemplified, but, as described above, when a featuresuch as a material or a physical property of the measurement target X isdetermined, composition amounts of various components contained in themeasurement target X, for example, each component amount of a firstcomponent, a second component, and a third component may be analyzed asan index value.

These index values are calculated for each of the pixels in the targetrange and recorded in the terminal storage section 34 as pixel analysisdata. For example, when a pixel A and a pixel B are included in thetarget range, and r=r_(A), g=g_(A), and b=b_(A) are calculated as RGBvalues that are index values for color determination with respect to thepixel A, and r=r_(B), g=g_(B), and b=b_(B) are calculated for the pixelB, pixel analysis data D_(A) (r_(A), g_(A), b_(A)) for the pixel A andpixel analysis data D_(B) (r_(B), g_(B), b_(B)) for the pixel B arecalculated and are stored in the terminal storage section 34 inassociation with position information indicating pixel positions.

Next, the classification section 355C of the analysis section 355determines whether or not the pixel analysis data or the feature vectorof each pixel in the target range is classified as the same system (stepS6). That is, it is determined whether or not the target range is arange including a configuration having the same feature. For example, inthe present embodiment, it is determined whether or not the target rangehas the same system color.

When NO is determined in step S6, the classification section 355Cclassifies the pixel analysis data into a plurality of groups (step S7).

For example, in the present embodiment, the pixel analysis data isclassified into a plurality of color groups of the same system colorsuch as a red system, a blue system, and a green system. In this case,the classification section 355C calculates, for example, a distance (forexample, a Mahalanobis distance or a Euclidean distance) of the pixelanalysis data of each pixel from color reference data, and sets aplurality of pieces of pixel analysis data having the farthest distanceas reference data. For each piece of the pixel analysis data, referencedata having the smallest distance is specified among the pieces ofreference data, and the pixel analysis data is classified as a group ofthe specified reference data.

The reference data may be set in advance. For example, in the presentembodiment, red reference data for red, green reference data for green,and blue reference data for blue may be set, and each piece of the pixelanalysis data may be classified based on a distance from the referencedata to the pixel analysis data.

Thereafter, the boundary setting section 355D sets the boundary in eachof the classified groups (step S8). For example, the boundary settingsection 355D forms a boundary surface that passes through a midpoint ofa line segment connecting pixel analysis data in each group to pixelanalysis data having the closest distance in another group and separatesthe groups from each other. The boundary surface may be a planeperpendicular to the line segment passing through the midpoint, or maybe a curved surface passing through the midpoint.

After step S8 and when YES is determined in step S6, the featuredetermination section 356 determines a feature in the target range basedon each piece of the pixel analysis data (step S9).

When YES is determined in step S6, that is, when all the pixel analysisdata in the target range are within a predetermined distance and are notclassified into groups, a feature in the target range of the measurementtarget is determined based on an average of pixel feature data or arepresentative value (for example, reference data) of the pixel featuredata.

When NO is determined in step S6, the pixel analysis data is classifiedinto a plurality of groups through step S7 and step S8. In this case,the feature determination section 356 determines a feature for eachgroup based on the pixel feature data of each group. For example, afeature for a group is determined based on an average of the pixelanalysis data belonging to the group or a representative value (forexample, reference data) of the pixel analysis data belonging to thegroup.

Thereafter, the analysis image generation section 357 generates ananalysis image in which each piece of pixel feature data is plotted onthe three-axis coordinate system based on the analyzed index value (stepS10: analysis image generation step).

FIGS. 4 to 6 are diagrams illustrating examples of analysis images.

FIG. 4 illustrates an analysis image when a single target range isdesignated in step S2 and each piece of pixel analysis data in thetarget range is not classified as a group.

FIG. 5 illustrates an analysis image when two target ranges aredesignated in step S2 and each piece of pixel analysis data in thetarget range is not classified as a group.

FIG. 6 illustrates an analysis image when a single target range isdesignated in step S2 and each piece of pixel analysis data in thetarget range is not classified into two groups.

As illustrated in FIGS. 4 and 6, when there is a single target range,the analysis image generation section 357 generates an analysis image 50displaying plot points 52 in which pixel analysis data as a result ofmultivariate analysis is plotted on a three-axis coordinate system graph51 with each index value as an axis.

For example, in the present embodiment, R, G, and B are analyzed from aspectral spectrum by using principal component analysis. In this case,as illustrated in FIG. 4, each piece of pixel analysis data is plottedon the three-axis coordinate system graph 51 with R, G, and B as axes.As illustrated in FIGS. 4 to 6, the analysis image 50 may display ananalysis method or analysis components obtained through the analysis.

When a material or a physical property of the measurement target X isdetermined as a feature, four or more index values may be calculated.For example, when an ingredient contained in a dish is determined, aplurality of component amounts such as water, sugar, lipid, protein, andiron are calculated as index values. In this case, for example, athree-axis coordinate system may be generated based on three indexvalues designated by a user, and a total for each component amountincluded in the target range may be calculated, and the top threecomponents with the largest total may be specified as index valuesrepresented in the three-axis coordinate system.

The analysis image generation section 357 may generate the analysisimage 50 in which a color image 53 captured in step S1 is alsodisplayed. In this case, the target range 54 designated by the user issuperimposed on the color image 53. The analysis image generationsection 357 displays the features determined by the featuredetermination section 356 in the analysis image 50. For example, in thepresent embodiment, the plot points 52 of the pixel analysis data aredisplayed according to the color of the measurement target X determinedby the feature determination section 356. Consequently, the user canrecognize the color of the target range 54 of the measurement target Xwith the plot points 52 on the three-axis coordinate system graph 51.The determination result in the feature determination section 356 may besuperimposed and displayed on the target range 54 of the color image 53.

When a plurality of target ranges 54 are designated in step S2, theanalysis image generation section 357 displays target ranges 54A and 54Bdesignated by the user in the color image 53 as illustrated in FIG. 5.Here, one target range 54 will be referred to as a first target range54A, and the other target range will be referred to as a second targetrange 54B.

The analysis image generation section 357 displays, on the three-axiscoordinate system graph 51, first plot points 52A of first pixelanalysis data corresponding to the first target range 54A and secondplot points 52B of second pixel analysis data corresponding to thesecond target range 54B. In this case, the first plot points 52A and thesecond plot points 52B may have different display forms. For example,the first plot point 52A is displayed as a “round” point and the secondplot point 52B is displayed as a “square” point. When a featuredetermination result using the first pixel analysis data and a featuredetermination result using the second pixel analysis data in the featuredetermination section 356 are different from each other, the first plotpoint 52A and the second plot point 52B may be displayed according tothe respective features. For example, when the first target range 54A isdetermined as being red and the second target range 54B is determined asbeing blue, the first plot point 52A may be displayed red and the secondplot point 52B may be displayed blue. In this case, each plot point anda corresponding target range may be displayed, for example, the firsttarget range 54A is displayed in the same color frame as that of thefirst plot point 52A, the second target range 54B is displayed in thesame color frame as that of the second plot point 52B.

In step S2, when the single target range 54 is designated and pixelanalysis data belonging to the target range is classified into aplurality of groups, the analysis image generation section 357 displaysthe plot points 52 corresponding to the respective groups on thethree-axis coordinate system graph in different display forms asillustrated in FIG. 6. For example, the example illustrated in FIG. 6 isan example in which the pixel analysis data is classified into twogroups, and third plot points 52C belonging to one group and fourth plotpoints 52D belonging to the other group are displayed. The analysisimage generation section 357 displays a boundary image 55 set by theboundary setting section 355D on the three-axis coordinate system graph51.

The analysis image generation section 357 may clearly indicate thetarget range 54 of the color image 53 such that the pixels correspondingto the third plot point 52C and the pixels corresponding to the fourthplot point 52D are displayed in different display forms. For example, inFIG. 6, when the feature determination section 356 determines that acolor feature is red for the group of the third plot point 52C and acolor feature is blue for the group of the fourth plot point 52D, eachpixel is replaced with a corresponding representative color. That is,the pixels corresponding to the third plot point 52C in the target range54 are displayed in the color image 53 in a high-brightness reddishrepresentative color (for example, R=255, G=0, and B=0) having apredetermined value or more. Similarly, the pixels corresponding to thefourth plot point 52D in the target range 54 are displayed in ahigh-brightness bluish representative color (for example, R=0, G=0, andB=255) having a predetermined value or more.

After step S10, the display control section 351 displays the generatedanalysis image 50 as illustrated in FIGS. 4 to 6 on the display 31 (stepS11).

The selection acquisition section 358 determines whether or not apredetermined plot point 52 in the three-axis coordinate system graph 51has been selected through the user's input operation (step S12).

When YES is determined in step S12, the selection acquisition section358 acquires the selected plot point 52 as a selected point. The displaycontrol section 351 displays a pixel corresponding to the selected pointas a corresponding pixel 56 (step S13).

FIG. 7 is a diagram illustrating an example of the analysis image 50displayed on the display after the process in step S13. For example,when one of the plot points 52 is selected as a selected point 52E inthe analysis image 50, the display control section 351 displays a pixelin the target range 54 corresponding to the selected point 52E as thecorresponding pixel 56 as illustrated in FIG. 7. The corresponding pixel56 is a pixel in which the pixel corresponding to the selected plotpoint is visually clearly indicated, and, for example, the correspondingpixel is blinked and displayed or is displayed at a different brightnessvalue by a predetermined value or more relative to the surroundingpixels.

Advantageous Effects of Present Embodiment

The analysis system 1 of the present embodiment includes the measurementdevice 2 and the terminal device 3, and the terminal control section 35of the terminal device 3 reads and executes the image analysis programstored in the terminal storage section 34 and thus functions as thespectral image acquisition section 352, the range acquisition section354, the analysis section 355, and the analysis image generation section357. The spectral image acquisition section 352 acquires spectral imagesfor a plurality of spectral wavelengths, obtained by imaging themeasurement target X. The range acquisition section 354 acquires thetarget range 54 in each spectral image. The analysis section 355performs multivariate analysis of each pixel based on a gradation valuefor each spectral wavelength of the pixel in the target range 54. Theanalysis image generation section 357 generates an analysis imageincluding the analysis result of the multivariate analysis for eachpixel in the target range 54.

In such an analysis system 1, the analysis image generated by theanalysis image generation section 357 is displayed, and thus not only afeature of the target range 54 but also an analysis result from whichthe feature determination has been performed in the target range can bedisplayed to the user. That is, when a feature of the measurement targetX is determined, the user cannot determine whether or not the determinedfeature is correct simply by displaying a final feature determinationresult. In contrast, as in the present embodiment, the analysis image 50as analysis result that is the intermediate data for determining afeature is displayed, and thus the user can understand the analysisresult on which the determined feature is based, and can thus determinewhether or not feature determination is possible.

In the present embodiment, the analysis section 355 functions as theindex calculation section 355B, and calculates a plurality of indexvalues indicating a feature of each pixel based on a gradation value foreach spectral wavelength of the pixel. The analysis image generationsection 357 generates the analysis image 50 in which the plot points 52indicating an analysis result (pixel analysis data) for each pixel areplotted on the three-axis coordinate system graph 51 configured withcoordinate axes corresponding to respective index values.

As described above, the index values calculated from the analysis resultof the multivariate analysis are displayed on the coordinates, and thusthe user can easily check the analysis result and determine the qualityof the feature determination result. That is, when the plot points 52are discrete in the three-axis coordinate system, even though thefeature determination section 356 determines a feature of the targetrange 54, reliability thereof is low, and, conversely, when the plotpoints 52 are close to each other, it may be determined that thereliability of a feature determined by the feature determination section356 is high.

In the present embodiment, the analysis section 355 functions as theclassification section 355C, and further classifies each piece of pixelanalysis data into a plurality of groups based on the pixel analysisdata for each pixel in the target range 54. The analysis imagegeneration section 357 displays the plot points 52 in different displayforms for each of the classified groups.

Consequently, even when there are a plurality of portions havingdifferent features in the target range, groups corresponding to therespective feature can be set, and the plot points 52C and 52D aredisplayed in different display forms for each group as illustrated inFIG. 6. Therefore, the user checks such an analysis image 50, and canthus easily understand respective pieces of pixel analysis datacorresponding to a plurality of features determined by the featuredetermination section 356 and properly determine the quality of eachfeature determination.

In the present embodiment, the analysis image generation section 357generates the analysis image 50 displaying the boundary image 55indicating a boundary between different groups.

That is, the analysis section 355 also functions as the boundary settingsection 355D, and sets the boundary between the groups when each pieceof pixel target data is classified into a plurality of groups by theclassification section 355C. The analysis image generation section 357displays the boundary image 55 corresponding to the set boundary on thethree-axis coordinate system graph 51.

Consequently, the user can easily understand by what threshold value thepixel analysis data is classified when a plurality of features aredetermined from the target range 54.

In the present embodiment, the terminal control section 35 of theterminal device 3 functions as the feature determination section 356.The feature determination section 356 determines a feature in the targetrange 54 based on an analysis result of multivariate analysis in theanalysis section 355. The analysis image generation section 357generates the analysis image 50 including the analysis result in theanalysis section 355 and the feature determined by the featuredetermination section 356. For example, in the present embodiment, theanalysis image 50 including the features of the target range is obtainedby displaying each plot point 52 in the color determined for the targetrange 54.

Consequently, the user can understand, from the analysis image 50, theplot points 52 corresponding to the pixel target data as a result of themultivariate analysis for each pixel of the target range 54 and thefeatures of the target range determined through the multivariateanalysis result.

In the present embodiment, the terminal control section 35 of theterminal device 3 also functions as the color image acquisition section353, and the color image acquisition section 353 acquires a color imageof the measurement target X. The analysis image generation section 357generates an analysis image including the three-axis coordinate systemgraph 51 and the color image 53 in which the target range 54 isdisplayed.

Consequently, the user can easily recognize which target range 54 of thecolor image 53 corresponds to data of each plot point 52 on thethree-axis coordinate system graph 51.

In the present embodiment, the terminal control section of the terminaldevice 3 functions as the display control section 351 and the selectionacquisition section 358. The display control section 351 displays theanalysis image 50 on the display 31. The selection acquisition section358 acquires the selected plot point 52 as the selected point 52E whenreceiving the user's operation for selecting a predetermined plot point52 for the analysis image 50 displayed on the display 31. The displaycontrol section 351 displays a pixel corresponding to the selected point52E in the color image 53 as the corresponding pixel 56.

Consequently, the user can individually understand which pixel in thecolor image 53 corresponds to data of the plot point 52 displayed on thethree-axis coordinate system graph 51.

Modifications of Embodiment

The present disclosure is not limited to the above-described embodiment,and modifications, improvements, and the like within the scope in whichthe object of the present disclosure can be achieved are included in thepresent disclosure.

Modification Example 1

In the above embodiment, an example has been described in which thecolor imaging section 22 continuously captures a color image in apredetermined cycle and continuously updates the color image displayedon the display 31 to the captured image as a real-time image, but thepresent disclosure is not limited thereto.

For example, capturing of a color image in the color imaging section 22and capturing of spectral images for a plurality of spectral wavelengthsin the spectral imaging section may be simultaneously performed, andthese images may be recorded in the terminal storage section 34 or thestorage section 251. In this case, the user can read the image stored inthe terminal storage section 34 and display the image on the display 31at any timing, and can designate the target range 54 for the displayedcolor image. Since the spectral image is also captured at the time ofcapturing the color image, it is not necessary to capture the spectralimage after the range acquisition section 354 acquires the target range54, and the spectral image corresponding to the color image may be readfrom the terminal storage section 34 or the storage section 251, and thesame analysis process as that in the above embodiment may be performed.

Modification Example 2

In the above embodiment, an example has been described in which, when aplurality of target ranges 54 are set by the user, the analysis imagegeneration section 357 displays the first plot point 52A and the secondplot point 52B on the three-axis coordinate system graph 51 asillustrated in FIG. 5. In this case, as illustrated in FIG. 6, theanalysis image generation section 357 may display the boundary image 55between the group of the first plot point 52A and the group of thesecond plot point 52B.

That is, the boundary setting section 355D may set a boundary based oneach piece of first pixel analysis data belonging to the first targetrange 54A and each piece of second pixel analysis data belonging to thesecond target range 54B, and the analysis image generation section 357may display the set boundary on the three-axis coordinate system graph51.

Modification Example 3

In the above embodiment, the selection acquisition section 358 acquiresany plot point 52 on the three-axis coordinate system graph 51 as theselected point 52E, but the present disclosure is not limited thereto.For example, the selection acquisition section 358 may acquire a pixeldesignated by the user as a selected point in the target range 54 of thecolor image 53. In this case, the display control section 351 may readpixel analysis data corresponding to the selected point from theterminal storage section 34, and display the corresponding plot point 52in the three-axis coordinate system graph 51 as a target plot point tobe visually clearly indicated.

Modification Example 4

In the above embodiment, when a plurality of target ranges 54 aredesignated and pixel analysis data included in each target range 54 isclassified into a plurality of groups, as illustrated in FIG. 5, theplot points 52 displayed on the three-axis coordinate system graph 51may be displayed in different display forms for the respective groups.That is, the plot points 52 may be displayed in different display formsfor respective target ranges and respective groups.

Modification Example 5

In the above embodiment, the feature determination process is performedby the feature determination section 356, but the feature determinationsection 356 may not be provided. In this case, a feature of themeasurement target X is not displayed in the analysis image 50, and theuser himself/herself determines the feature of the measurement target Xin the target range 54 based on an analysis result. In this case, asdescribed above, since an analysis result for each pixel in the targetrange is displayed in the analysis image 50, the user can easilydetermine the feature in the target range.

Modification Example 6

In the above embodiment, an example has been described in which theanalysis image 50 includes the three-axis coordinate system graph 51corresponding to the three index values, but the present disclosure isnot limited thereto. For example, a two-axis coordinate system graphcorresponding to two index values may be included, and the analysisimage 50 using a coordinate system of four or more axes may bedisplayed. An index value of each piece of pixel analysis data may bedisplayed as a numerical value in a text or table form.

An example has been described in which a color image of the measurementtarget X that is an imaging target is displayed as an analysis image,but the color image may not be displayed as the analysis image.

Overview of Present Disclosure

An image analysis apparatus of a first aspect of the present disclosureincludes a spectral image acquisition section that acquires spectralimages for a plurality of wavelengths, obtained by imaging a measurementtarget; a range acquisition section that acquires a target range in eachof the spectral images; an analysis section that performs multivariateanalysis of each pixel based on a gradation value of the pixel for eachwavelength in the target range; and an analysis image generation sectionthat generates an analysis image including an analysis result of themultivariate analysis for each pixel in the target range.

Consequently, the analysis image generated by the analysis imagegeneration section is displayed, and thus it is possible to show a userwhat kind of analysis result was obtained in the target range. That is,when a feature of the measurement target is determined, the user cannotdetermine whether or not the determined feature is correct simply bydisplaying a final feature determination result. In contrast, in thepresent aspect, since it is possible to obtain an analysis imageindicating the analysis result which is intermediate data fordetermining the feature, the user can determine the feature of themeasurement target based on the analysis result.

In the image analysis apparatus of the present aspect, the analysissection may calculate a plurality of index values indicating a featureof each pixel based on the gradation value of the pixel for eachwavelength, and the analysis image generation section may generate theanalysis image in which plot points indicating the analysis result foreach pixel are plotted in a coordinate system configured with coordinateaxes corresponding to the respective index values.

As described above, since the index values calculated from the analysisresult of the multivariate analysis are displayed on the coordinates,the user can easily check the analysis result and thus easily determinea feature of the measurement target in the target range. That is, in thethree-axis coordinate system, when the plot points are discrete, it maybe determined that the measurement target includes various features, andconversely, when the plot points are close to each other, it may bedetermined that a feature of the measurement target is fixed.

In the image analysis apparatus of the present aspect, the analysissection may further classify each pixel into a plurality of groups basedon the analysis result for each pixel in the target range, and theanalysis image generation section may display the plot points indifferent display forms for the respective classified groups.

Consequently, even when there are a plurality of portions havingdifferent features in the target range, the plot points can be displayedin different display forms for the respective groups. Therefore, theuser can easily understand that a plurality of features of themeasurement target are included by checking such an analysis image.

In the image analysis apparatus of the present aspect, the analysisimage generation section may generate the analysis image in which aboundary image indicating a boundary between the different groups isdisplayed.

Consequently, the user can easily understand by what threshold valuepixel analysis data is classified when a plurality of features aredetermined from the target range.

The image analysis apparatus of the present aspect may further include afeature determination section that determines a feature in the targetrange based on the analysis result of the multivariate analysis, and theanalysis image generation section may generate the analysis imageincluding the analysis result and the feature.

Consequently, the user can understand, from the analysis image, thefeature of the target range determined through the multivariate analysisresult in addition to the result of the multivariate analysis for eachpixel of the target range. Therefore, the user does not have todetermine the feature of the target range by himself/herself based onthe analysis result. Since the analysis result of the multivariateanalysis is displayed in the analysis image, the user can check whetheror not a determination result of the feature determined by the featuredetermination section is correct based on the analysis result.

The image analysis apparatus of the present aspect may further include acolor image acquisition section that acquires a color image of themeasurement target, and the analysis image generation section maygenerate the analysis image including the color image in which thetarget range is displayed.

Consequently, the user can easily recognize which target range of thecolor image corresponds to data of each plot point indicating ananalysis result displayed in the analysis image.

In the image analysis apparatus of the present aspect, the analysissection may calculate a plurality of index values indicating a featureof each pixel based on the gradation value of the pixel for eachwavelength, and the analysis image generation section may generate theanalysis image in which plot points indicating the analysis result foreach pixel are plotted in a coordinate system configured with coordinateaxes corresponding to the respective index values, the image analysisapparatus may further include a display control section that displaysthe analysis image on a display section, and a selection acquisitionsection that, when receiving a user's operation for selecting apredetermined plot point on the analysis image displayed on the displaysection, acquires the selected plot point as a selected point, and thedisplay control section may display a pixel corresponding to theselected point in the color image as a corresponding pixel.

Consequently, the user can individually understand which pixel in thecolor image corresponds to data of the plot point plotted on thecoordinate system in the analysis image.

An image analysis method of a second aspect of the present disclosureincludes causing a computer to analyze spectral images for a pluralityof wavelengths, obtained by imaging a measurement target, in which thecomputer functions as a spectral image acquisition section, a rangeacquisition section, an analysis section, and an analysis imagegeneration section, the spectral image acquisition section executes aspectral image acquisition step of acquiring the spectral images for theplurality of wavelengths, obtained by imaging the measurement target,the range acquisition section executes a range acquisition step ofacquiring a target range in each of the spectral images, an analysissection executes an analysis step of performing multivariate analysis ofeach pixel based on a gradation value of the pixel for each wavelengthin the target range, and an analysis image generation section executesan analysis image generation step of generating an analysis imageincluding an analysis result of the multivariate analysis for each pixelin the target range.

Consequently, in the same manner as in the first aspect, the analysisimage generated by the analysis image generation section is displayed,and thus it is possible to show a user what kind of analysis result wasobtained in the target range.

An image analysis program of a third aspect of the present disclosure isa program that can be read and executed by a computer, and causes thecomputer to function as any one of the above image analysis apparatuses.

Consequently, in the same manner as in the first aspect, the analysisimage generated by the analysis image generation section is displayed,and thus it is possible to show a user what kind of analysis result wasobtained in the target range.

What is claimed is:
 1. An image analysis apparatus comprising: one ormore processors configured to execute (a) acquiring, from a measurementdevice, spectral images for a plurality of wavelengths, obtained byimaging a measurement target, (b) acquiring a target range in each ofthe spectral images, (c) performing multivariate analysis of each pixelbased on a gradation value of the pixel for each wavelength in thetarget range, (d) generating an analysis image including an analysisresult of the multivariate analysis for each pixel in the target range,and (e) storing the generated analysis image into a memory.
 2. The imageanalysis apparatus according to claim 1, wherein the one or moreprocessors are further configured to execute calculating a plurality ofindex values indicating a feature of each pixel based on the gradationvalue of the pixel for each wavelength in (c), and generating theanalysis image in which plot points indicating the analysis result foreach pixel are plotted in a coordinate system configured with coordinateaxes corresponding to the respective index values in (d).
 3. The imageanalysis apparatus according to claim 2, wherein the one or moreprocessors are further configured to execute classifying each pixel intoa plurality of groups based on the analysis result for each pixel in thetarget range in (c), and displaying the plot points in different displayforms for the respective classified groups in (d).
 4. The image analysisapparatus according to claim 3, wherein the one or more processors arefurther configured to execute generating the analysis image in which aboundary image indicating a boundary between the different groups isdisplayed in (d).
 5. The image analysis apparatus according to claim 1,wherein the one or more processors are further configured to executedetermining a feature in the target range based on the analysis resultof the multivariate analysis, and generating the analysis imageincluding the analysis result and the feature in (d).
 6. The imageanalysis apparatus according to claim 1, wherein the one or moreprocessors are further configured to execute acquiring a color image ofthe measurement target from the measurement device, and generating theanalysis image including the color image in which the target range isdisplayed in (d).
 7. The image analysis apparatus according to claim 6,the one or more processors are further configured to execute calculatinga plurality of index values indicating a feature of each pixel based onthe gradation value of the pixel for each wavelength in (c), generatingthe analysis image in which plot points indicating the analysis resultfor each pixel are plotted in a coordinate system configured withcoordinate axes corresponding to the respective index values in (d),displaying the analysis image on a display, when receiving a user'soperation for selecting a predetermined plot point on the analysis imagedisplayed on the display, acquiring the selected plot point as aselected point, and displaying a pixel corresponding to the selectedpoint in the color image as a corresponding pixel.
 8. A non-transitorycomputer-readable storage medium storing a program executed by one ormore processors and thus to cause the one or more processors to execute:(a) acquiring spectral images for a plurality of wavelengths, obtainedby imaging a measurement target; (b) acquiring a target range in each ofthe spectral images; (c) performing multivariate analysis of each pixelbased on a gradation value of the pixel for each wavelength in thetarget range; (d) generating an analysis image including an analysisresult of the multivariate analysis for each pixel in the target range;and (e) storing the generated analysis image into a memory.